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ES  Software  Industry  Challenges 

Embedded  Systems  Performance 

❖  ES  revolution  started  in  industry  rather  than  universities 
♦  Common  systems  engineering  problems  haven’t  been 
scientif  ically  addressed 

❖  Shift  from  Hardware  to  Software  (“softwareization”) 

❖  Dramatic  Increase  in  the  complexity  of  functionality 

♦  Number  of  lines  of  code  per  function  in  aircraft  systems 
was  10  in  1970,  now  1,000,000 

♦  Increase  in  observable,  controllable  parameters 

♦  Trend  to  interoperability  of  ES  in  networks 

V 

❖  Crowing  gap  between  software  size  and  developers’ 
productivity 
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Why  Worry  About  Performance? 

❖  Many  systems  experience  performance  problems  on 
initial  release 

❖  Problems  are  often  due  to  fundamental  architecture  or 
design  rather  than  ineff icient  code 

♦  Introduced  early  in  development 

♦  Not  discovered  until  late 

❖  “Tuning”  code  after  implementation 

♦  Disrupts  schedules  and  creates  negative  user  perceptions 

♦  Results  in  poorer  overall  performance  (than  building 
performance  into  architecture) 

♦  May  not  be  possible  to  achieve  requirements  with  tuning 

♦  Increases  costs 
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Value  of  Preventing  Problems 


US  Federal  Government  Software 


Used  But  Extensively 

Delivered  But  Never  Paid  for  But  Reworked  or 

Successfully  Used  Never  Delivered  Abandoned  Used  After  Changes  Used  As  Delivered 
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Lessons  from  History 

Modernizing  Telephone  Switch 
Software 

♦  Initial  implementation  of  object 
oriented  software  resulted  in 
signif icant  performance  problems 

♦  Many  00  telephony  systems  had  the 
same  performance  problems 
(Software  Performance  Antipattern) 

♦  Preventable  with  proper  tools 

♦  Risk  of  new  technology  and/or 
inexperienced  personnel 

♦  Problems  likely  to  occur  in  initial  MDE 
implementation  for  Embedded 
Systems 
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RTES/ Analyzer  Performance  Modeling 


❖  Automated  assessment  of  software  and  systems 
architecture  is  essential 

♦We  cannot  continue  to  build  RTES  with  yesterday’s 
methods 

❖  RTES/Analyzer  approach 

♦Model  interoperability 

Automated  translation  of  design  models  to  performance 
models 

>Model  solutions  translated  into  meaningful  results  for 
developers 

♦Adaptable,  extensible  evolution  of  tools 
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SPE  Balance 


Requirements 


❖  Quantitative  Assessment 

❖  Begins  early,  frequency  matches  system  criticality 

❖  Often  find  architecture  A  design  alternatives  with 
lower  resource  requirements 

❖  Select  cost-effective  performance  solutions  early 

❖  Right-size  the  platform 
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SPE  Models 


System  Models  Software  Prediction  Models 
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SPE  Model  Requirements 

❖  Low  overhead 

♦  use  the  simplest  possible  model  that  identifies  problems 

❖  Goals: 

♦  initially  distinguish  between  "good"  and  "bad" 

♦  later,  increase  precision  of  predictions 

♦  provide  decision  support 
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❖Tool  for 

performance 

engineers 

❖Established 

technology 

❖  Access  to 
source  code 
for  RAD 


L&S  Computer  Technology.  Inc.©201 1 


©2011  L&S  Computer  Technology,  Inc. 


3 


SSTC  -  Dr.  Connie  U.  Smith 


May  18,2011 


Additional  SPE  Topics 

❖  Performance  Principles 

❖  Performance  Measurement 

❖  Performance  Patterns  &  Antipatterns 

❖  Architecture  Assessment:  PASASM 

❖  Business  Case  for  SPE 

❖  SPE  Process 
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Performance  Solutions 

A  Practical  Glide  to  Creating 
Responsive,  Scalable  Software 


CONNIE  U.  SMITH 
LLOYD  G.  WILLIAMS 


BOOCH 

JACOBSON 

RUKBflUCH 


Part  2:  Automating  the  Model-Driven 
Analysis 


V 
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UML  Design  Models  ->  Performance  Models 


j 
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Model  Interchange 
Formats  (MIFs) 
streamline  model 
interoperability 
process 
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MIF  Approach 

General  approach  to  be  used  by  a  wide  variety  of  tools 

♦  El  A  EDIF/CDIF  paradigm 

♦  Meta-model  of  information  requirements 

♦  Transfer  format  based  on  meta-model 

❖  XML  implementation 

♦  Meta-model  ->  schema,  transfer  format  in  XML 

♦  Relatively  easy  to  create 

❖  Common  interface 

♦  No  need  for  n2  customized  interfaces  between  tools 

♦  Import/export  can  be  external  to  tools  with  file  interfaces 
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Our  Model  Interchange  Research  Results 

❖  Design  tools  to  software  performance  models  (S-PMIF) 

❖  System  performance  models  (PMIF) 

❖  Model  solutions 

♦  Experiments  (Ex-SE) 

♦  Output  metrics  desired  from  experiments  (Output-SE) 

♦  Transformation  from  output  to  tables  and  charts 
(Results-SE) 
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Vision:  Developers  Do  Robust  Engineering 

❖  Explore  options  using  familiar  tools  &  notations  (UML, 
Eclipse) 

❖  Select  candidate  designs  for  exploration 

❖  RTES/Analyzer 

♦  Select  metrics 

♦  Specify  analysis  conditions  and  select  tools 

♦  Quantitative  predictions  from  multiple  tools 

♦  Environment  invokes  analysis  tool(s),  collects  output, 
prepares  results  in  user-f riendly  format 

♦  Identify  performance  antipatterns 

❖  Bring  in  performance  specialists  for  serious  problems 
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Robust  Enaineeringof 
Large  Distributed  RTES 
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Concept  Diagram 


Objective 

•Robust  Framework  for  automatic  performance  assessment  of  RTES 

•Translate  designs  to  performance  models 

•Define  and  execute  experiments 

•Convert  output  metrics  to  meaningful  results 

•Compare  results  from  multiple  tools 

•Ability  to  extend  Framework  with  new  analysis  capabilities  for 

developers 

•Automated  studies  (scalability,  sizing,  sensitivity,  etc.) 

•Identify  problematic  design  features  and  performance  antipatterns 


Approach 

Based  on  Performance  Model  Interchange  Formats  (PMIF  and  S- 
3MIF)  and  tool  import  and  export  interfaces 
Interoperability  of  heterogeneous  software  design  and  performance 
analysis  tools 
Develop  Prototypes 

Representative  tools  and  services  for  end-to-end  analysis  from  design 
:o  meaningful  results 

•Real-time  embedded  system  (RTES)  focus 
•Ability  to  evaluate  partial  designs  &  combine  designs 
•Demonstration  -  representative  DoD  RTES 
•Marketing,  sales,  commercialization  plan 


Impact/Milestones 

•FY09 

•Enabling  technology  complete 
•Architecture  complete 

•Improved  analysis  capabilities  can  cut  up  to  95%  of  time  required  for 

(manual)  performance  analysis  of  designs 

•Automatically  keep  design  and  performance  models  in  sync 

•Performance  models  keep  pace  with  design  changes 

•Eliminates  manual  comparison  and  re-creation  of  models 

•Ease  of  use  increases  likelihood  of  conducting  performance  studies 

early  in  lifecycle 

•Result:  Better  performing  systems  with  optimally  sized  networks  and 
platforms  reduces  hardware  costs 


Technology  &  Target  Market:  Analysts  and  Developers  of  Real-Time  Embedded  Systems  (RTES) 


Case  Study 


V 
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Robot  Controller  SEI  Model  Problem 


❖  Main  computer  generates  work  orders 

❖  Decomposed  into  subwork  orders  to  axis  computer(s) 

❖  Interpreted  by  device  drivers  for  movement  of  robot 
arms 
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Controller  Design 


ckxk2000 

❖  Movement  planner  cannot  find  repository  empty 

❖  Planners  cannot  miss  deadlines  at  end  of  period 
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Component  Architecture  ->  Performance  Models 


Software  model:  Construction  & 
Composition  Language  (CCL) 


Automated  Transformation  to 
S-PMIF  Performance  Model 


Model  Solutions 
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S-PMIF  MetaModel 


Workload  Platform 
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S-PMIF  Excerpt 

<Perf ormanceScenario  InterarrivalTime="450 . 0" 

MainEG="tra jectoryPlanner . go"  Priority=" 4 " 

ScenarioId="tra jectoryPlanner . go" 

ScenarioName="tra jectoryPlanner . go"  SWmodelf ilename="icm"> 

<ExecutionGraph  EGId="tra jectoryPlanner . go" 

EGname="tra jectoryPlanner . go" 

StartNode="N_tra jectoryPlanner . go"> 

<BasicNode  NodeId="N_tra jectoryPlanner . go" 

NodeName="N_tra jectoryPlanner . go"> 

<SWResourceReguirement  SWResourceId="R_CPU" 

Units Of Service=" 8 9 . 665066 "/> 

</BasicNode> 

<SynchronizationNode 

NodeId="N_tra jectoryPlanner . read_posit ionMonitor . read" 
NodeName="N_tra jectoryPlanner . read_posit ionMonitor . read" 
myType="SynchronousCall"  partnerId="N_positionMonitor . read" 
part nerScenario= "posit ionMonitor . read" /> 

_ L&S  Computer  Technology,  Inc.©201 1 _ 28_ 
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Performance  Analysis 

Best  and  worst  case  analysis 

Simple  model  and  advanced  model  with  synchronization 
*  Multiple  tools 

♦  Worst  case  latency  -  PSK  performance-reasoning 
framework  on  linear  sequence  of  actions 

>  MAST  tool  -  RMA  technique 

>  Discrete  event  simulator 

♦  SPE-ED  tool 
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Software  Performance  Models 


Simple:  scenario 
per  event  source  (4) 


E_trajectoryPlanner.go 


Time,  no  contention:  1 1 2.65 


NJrajedory 
Parmer  go 


89.67 


<179.7 

<  269.8 

<  359.9 
<450.0 
>=  450 .0 


N  position  3.06 


N_ronos»- 

to'y.acxess 


19.92 


Advanced:  scenario 
per  thread  with 
synchronization  (9) 


Tick_450 


T  rajecory 
Planner.go 


posMonitor.read 

Q 


repository.access 
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Model  Results 


Transaction 

Ek»st 

Avorags 

Worst 

RMA  Analytic 

dock  130  tick 

15  04 

66  04 

dock*  50  tick 

11?  65 

?6?  /7| 

dock  150  tick 

60  0? 

79  94 ' 

dock?tXJ0  t>ck 

0  3? 

?78  14 

□E  Simulation 

dockl30  tick 

15  04 

33  71 

75  08 

dock-450  tick 

?47  73 

?!>9  49 

?6?  83 

dock  150  tick 

60  0? 

60  00 

60  04 

ctoc*2000.i»ck 

0.32 

103.08 

27820 

SPE  ED  Results 

dock  130.  tick 

15  04 

3378 

99  071 

doc  k4 50. tick 

112  65 

25967 

262  77 

dock  150.  tick 

60.02 

60.02 

60.02 1 

dock2DD0  lick 

0.32 

71.61 

278.14 
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Results 


❖  Simulation  solutions  comparable,  not  exact 

♦  DE  simulation  does  not  include  contention 

♦  In  best  case,  response  to  clock450.tick  preempted  twice 
by  clockl50.tick  ->  higher  response  time  than  no  contention 
best  case 

❖  Simple,  best  case  is  optimistic 

♦  Identifies  problems  that  must  be  corrected 

♦  Then  proceed  to  more  precise  evaluations 
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Option 

❖  Replace  X  and  Y  controllers  with  controllers  that  also 
provide  position  feedback  to  position  monitor 

♦  Simple  model:  changes  Clockl50.tick  to  make  +2  calls 

♦  Ad\/ar\czd  model:  changes  ControllerX  and  ControllerY 
threads  to  make  asynchronous  calls  to 
PositionMonitor.input 
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Proof  of  Concept 

❖  Demonstrates  viability  of  model  interchange  approach 

❖  Builds  on  work  in  component-based  systems,  SPE,  and 
model  interchange 

❖  Helpful  to  compare  solutions  from  different  software 
performance  modeling  tools 

❖  Automation  of  steps  simplif ies  performance  assessment 
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Revised  Results 


Transaction 

Best 

Average 

Worst 

RMA  Analytic 

dock130.bck 

15.04 

124.06 

dock450.bck 

112  65 

496.91 

dock  150.  tick 

8603 

109.02 

dock20Q0.t»ck 

0.32 

431.24 

OE  Simulation 

dock130.bck 

15  04 

52.18 

115  99 

dock450.bck 

314  80 

347  63 

431.04 

dock150.bck 

8603 

89  57 

105.99 

dock20D0  t*ck 

1619 

220  18 

431  36 

SPE  ED  Rosuit* 

dock  130  be* 

15  04 

46  51 

20616 

docMSfl  befc 

11265 

305  60 

31 7  88 

dock  150  befc 

86  03 

90  08 

192  65 

dock2COO  lek 

0  32 

128  68 

413  30 

Worst-case  times  differ: 

SPE*  ED  computed  average  time  for  all  calls  to  positionMonitor.input 
RMA  distinguishes  between  calls  from  different  “clocks”  -  each  has 
different  response  time  due  to  pre-emption 
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Case  Study  Conclusions 

❖  S-PMIF  transformations  can  be  procedural  (custom 
code)  or  declarative  Model  to  Model  (M2M) 
transformations 

❖  Enables  performance  analysis  of  CCL  specif ications  with 
additional  analysis  tools  without  special  integration 
efforts 

❖  Demonstrates  viability  and  ease  of  using  S-PMIF  with 
multiple  design  notations  in  addition  to  UML 
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RTES/Analyzer:  Sample  User  Interface 


V 
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UI  Demonstration 


RTES  Analyzer 


AriAly/4*  «wo»tw.*r«* 
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UI  Demonstration 

❖  Demonstrates  ease  of  use  for  developers 

❖  Selection  of  designs  and  experiments 

❖  Meaningful  results 

❖  Flashbuilder  foundation  for  Phase  2  implementation 
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SPE  ED  ->  RTES/Analyzer 


♦  Users  are  performance  experts 

♦  Primarily  IT  systems 

RTES/Analyzer 

♦  Target  developers  as  users 

♦  Focus  on  Real-Time  &  Embedded  System  market  sector 
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RT/Analyzer  Addresses  Future  Needs 

❖  Cost 

♦  Ability  to  predict  performance  of  designs  reduces  cost  of 
re-work  due  to  late  discovery  of  problems 

♦  Up  to  100  times  more  expensive  to  fix  it  later 

❖  Quality 

♦  Systems  meet  performance  requirements 

❖  Automated  Analysis 

♦  RT/Analyzer  early  detection  of  problems,  performance  ranking 
of  solutions 

♦  Less  expertise  and  shorter  time  for  analysis 

❖  Productivity 

♦  Quicker  to  build-in  performance 

♦  Resources  can  be  devoted  to  development  rather  than  re-work 
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Status 


❖  RTES/Analyzer  architecture  and  enabling  technology  are 
positioned  for  future  development 

❖  SBIR  Phase  II  funding  approved 

❖  Developing  prototype  RTES/Analyzer  to  demonstrate  the 
viability  of  automatic  generation  and  evaluation  of 
performance  models,  and  presentation  of  quantitative 
results  useful  for  developers 

❖  Seeking  comprehensive  case  study  data 

❖  Seeking  partners  to  create  commercial  products 
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Summary 


Embedded  Systems  Modeling 

Software  Performance 
Engineering  (SPE)  Overview 

Automating  the  Model-Driven 
Analysis 

Proof  of  Concept:  Component- 
based  RTES 
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Questions? 


cusmith@spe-ed.com 

http://www.spe-ed.com/ 
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